A liar will not be believed, even when he speaks the truth.
– Aesop
Not even the Bocca della Verità can tell if a Chinese medical study is fabricated.
So I offer this short entry on an issue that arose this week.
It has long been noted that there is a discrepancy in outcomes in clinical trials of Traditional Chinese Pseudo-Medicine (TCMP) between those studies that have been done in China and those done in the US and Europe.
99% of acupunctures studies in China for acupuncture are positive and:
No trial published in China or Russia/USSR found a test treatment to be ineffective.
This odd result has always been ascribed to a combination of culture bias and, perhaps, poor methodologies.
The method of randomisation was often inappropriately described. Blinding was used in only 15% of trials. Only a few studies had sample sizes of 300 subjects or more. Many trials used as a control another Chinese medicine treatment whose effectiveness had often not been evaluated by randomised controlled trials. Most trials focused on short term or intermediate rather than long term outcomes. Most trials did not report data on compliance and completeness of follow up. Effectiveness was rarely quantitatively expressed and reported Intention to treat analysis was never mentioned. Over half did not report data on baseline characteristics or on side effects. Many trials were published as short reports. Most trials claimed that the tested treatments were effective, indicating that publication bias may be common; a funnel plot of the 49 trials of acupuncture in the treatment of stroke confirmed selective publication of positive trials in the area, suggesting that acupuncture may not be more effective than the control treatments
As we have discussed at length in this blog, the poorer the methodology, the more likely a pseudo-medical study will have a positive result.
But the issue is not cultural bias or badly done studies. Nope.
Chinese researchers just make shit up.
An evaluation by the Chinese government found that:
more than 80 percent of clinical data is “fabricated,”
It seems that making up data is de rigueur in China:
“Clinical data fabrication was an open secret even before the inspection,” the paper quoted an unnamed hospital chief as saying.
Academic ethics is also an underdeveloped field in China, leading to an academic culture that is accepting of manipulation of data.
I don’t think that the 80 percent figure is overstated. I grew up a science fiction fan and a follower of Sturgeon’s law: “ninety percent of everything is crap.” It looks like Chinese medical researchers use it as a commandment rather than a warning.
The review does not specifically mention if TCPM studies were included and my reading suggests that the investigation applies to standard biomedical research. But given the fantastical nature of TCPM and the vanishingly small (on order of gravitation wave detection, one-ten-thousandth the diameter of a proton (10-19 meter)) prior plausibility of TCPM and likelihood that most of TCPM is Oakland, I would wonder if 80% is a gross underestimate.
Which presents an interesting and difficult problem. At what point can you no longer trust the veracity of the literature? 1.97% of researchers admit to making up data and more have other nefarious practices. Worrisome but not as difficult as a culture of fabrication. A single researcher will do less damage than a whole society. A single fake paper is likely to be swamped by the preponderance of the literature. But what if the entire literature is built on lies? Then what?
Evaluating the medical literature as a practicing physician, I would assume that if a researcher made up his data for one study, then all his data and subsequent bloviating would be suspect. Like Wakefield. I would assume anything Wakefield says would be a lie until I could prove otherwise. And that goes against my default position to assume that people are usually not deliberately lying to me. I was raised to think a person is only as good as their word. Of course outside of the medical literature lying is not always considered a detriment. For the electoral college 80% is evidently just fine.
If patients put their health, wealth, and life in my hands, I have to know that the clinical trials I am basing my decisions on are as close to true as possible. Truthiness is not good enough.
Can we automatically assume that all the biomedical research coming out of China is fabricated? How can we tell? We can’t. And I would be especially skeptical of clinical trials of TCPM. It would also be interesting to see the results of the almost too numerous to count acupuncture meta-analyses if they were re-analyzed after removing all the Chinese studies.
Unfortunately when it comes to the medical literature from China, since the information can’t be verified you have to assume it cannot be trusted and so no longer counts. It has to be considered persona non grata unless it can demonstrate otherwise.
I assume true believers in TCPM will ignore the findings and point to the malign influence of big pharma on medical research, as if that makes the lies out of China just peachy.
Dr. Cummings over at the BMJ Blogs was recently upset that acupuncture is considered by Wikipedia to be a pseudo-science. Acupuncture/TCPM literature is likely far worse, the lying lies of liars.